Confidence Interval Estimation in System Dynamics Models: Bootstrapping vs. Likelihood Ratio Method

In this paper we discuss confidence interval estimation for system dynamics models. Confidence interval estimation is important because without confidence intervals, we cannot determine whether an estimated parameter value is significantly different from 0 or any other value, and therefore we cannot determine how much confidence to place in the estimate. We compare two methods for confidence interval estimation. The first, the “likelihood ratio method,” is based on maximum likelihood estimation. This method has been used in the system dynamics literature and is built in to some popular software packages. It is computationally efficient but requires strong assumptions about the model and data. These assumptions are frequently violated by the autocorrelation, endogeneity of explanatory variables and heteroskedasticity properties of dynamic models. The second method is called “bootstrapping.” Bootstrapping requires more computation but does not impose strong assumptions on the model or data. We describe the methods and illustrate them with a series of applications from actual modeling projects. Considering the empirical results presented in the paper and the fact that the bootstrapping method requires less assumptions, we suggest that bootstrapping is a better tool for confidence interval estimation in system dynamics models. Introduction Statistical parameter estimation is increasingly used in system dynamics. Popular simulation software (e.g., Vensim, Powersim) includes tools for model calibration, allowing modelers to estimate parameters easily. These packages enable modelers to use robust methods that can deal appropriately with nonlinear feedback systems, systems for which simple methods such as ordinary least squares are not suitable because they routinely involve autocorrelation, endogeneity of explanatory variables, heteroskedasticity, and other violations of the maintained hypotheses of basic methods. While tools for estimating parameters have improved and are now easy to use in system dynamics software, less attention has been paid to the problem of finding confidence intervals around the estimated parameters. Violations of standard assumptions such as identically and independently distributed (iid) normal error terms make the problem of estimating the confidence intervals around best-fit parameters at least as difficult as finding the best estimates themselves. Without confidence intervals (or equivalently, hypothesis testing) we cannot determine whether an estimated parameter value is significantly different from 0 or any other value, and therefore we cannot determine how much confidence to place in the estimate. * I’m indebted to John Sterman for supporting this research. Jeroen Struben, JoAnne Yates and Hazhir Rahmandad provided helpful comments. All errors are mine.